Amended oper docs for RSE and Gatekeeper.
Made CAPI ansible playbooks work again.
Made browser based flow for SC API.
Still need to finish sc migration (rename of core/global etc. will break anything with list_service_types etc, client, datalink? dm api?, astroquery? gatekeeper? - check all)
Fixing elasticsearch (permissions fault on /data, needed to add +w to elasticsearch group on each pvc by creating a debug pod that consumes this PVC) -- Restarted node
Fixing grafana (500, database corrupted) -- Had to export/import the database (https://www.prisma.io/dataguide/sqlite/importing-and-exporting-data-in-sqlite) -- watch out for ROLLBACK (not COMMIT) at end of exported sql (otherwise 0 byte import) -- Had to reload pod
Rucio/harbor down -- Restarted harbor core/database pods -- Restarted rucio
Elasticsearch back down again -- Restarted node -- Restarted all pods after 0 reporting can't get quorum
Harbor down -- 502s, restarted registry pods
Rucio app postgres + metadata down -- adjusted docker reg url: registry.gitlab.com/ska-telescope/src/src-mm/ska-src-mm-rucio-ivoa-integrations/postgres-metadata:bitnami-dachs -- killed database pods and let them respawn
Mount issues:
Must be a better way rather than restart of node.
Go to es master running on 9l..
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Normal Scheduled 4m19s default-scheduler Successfully assigned monitoring/elasticsearch-master-2 to srcnet-workload-1-md-0-9nlmq │
│ Warning FailedMount 2m17s kubelet Unable to attach or mount volumes: unmounted volumes=[elasticsearch-master], unattached volumes=[kube-api-access-25bpb elasticsearch │
│ master esconfig[]: timed out waiting for the condition │
│ Warning FailedMount 19s (x2 over 2m20s) kubelet MountVolume.SetUp failed for volume "pvc-f2c087c7-c507-4ed1-95f8-bf41a8311c4a" : rpc error: code = DeadlineExceeded desc = context d │
│ eadline exceeded │
│ Warning FailedMount 0s kubelet Unable to attach or mount volumes: unmounted volumes=[elasticsearch-master], unattached volumes=[elasticsearch-master esconfig kube- │
│ pi-access-25bpb[]: timed out waiting for the condition Off to corresponding Manila nodeplugin pod:
│ cephfs-nodeplugin E0627 09:25:34.637039 1 driver.go:332] [ID:24356] GRPC error: rpc error: code = Internal desc = an error (exit status 32) occurred while running umount args: [/var/lib/kube │
│ let/pods/23d4368d-a520-40cb-ba25-c3ae8451f11b/volumes/kubernetes.io~csi/pvc-3fcfd5d7-b918-4512-a205-fd3c067d8504/mount] stderr: umount: /var/lib/kubelet/pods/23d4368d-a520-40cb-ba25-c3ae8451f11b/v │
│ olumes/kubernetes.io~csi/pvc-3fcfd5d7-b918-4512-a205-fd3c067d8504/mount: target is busy.Tried to restarting the manila controller pod on this node. Didn't work.
Trying to umount e.g.
root@srcnet-workload-1-md-0-9nlmq:/var/lib/kubelet/pods/cd68ed72-6f40-4fdc-8205-7896cfcb0b7a/volumes/kubernetes.io~csi/pvc-f2c087c7-c507-4ed1-95f8-bf41a8311c4a# sudo umount /var/lib/kubelet/pods/ee7d315d-cc7c-4557-b38a-b496e8a6e423/volumes/kubernetes.io~csi/pvc-f2c087c7-c507-4ed1-95f8-bf41a8311c4a/mountHangs (the directory itself is empty).
Restarted kubelet:
ubuntu@srcnet-workload-1-md-0-9nlmq:~$ sudo systemctl restart kubeletand deleted pod.
Seemed to work, but now lots stuck in Pending.
Had to do a shutdown -r now